ORGANIZATION 


NATO  Code  of  Best  Practice 
(COBP)  for  C2  Assessment 


ationships  and  dynamics 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1 .  REPORT  DATE  2.  REPORT  TYPE 

00  DEC  2003  N/A 

3.  DATES  COVERED 

4.  TITLE  AND  SUBTITLE 

Assessment  Participants,  Relationships  and  Dynamics 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

NC3A  AC3IA  Branch  ORFSD  P  O  Box  174  2500  CD  The  Hague  THE 
NETHERLANDS 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

See  also  ADM001657.,  The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

APlSITT?  apt 

1 8 .  NUMBER  1 9a.  NAME  OF 

r\  T7  DACttnC:  T?1nQDrY\TQrRT  P  DThT?  QCtM 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  jjjj 

unclassified  unclassified  unclassified 

57 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Notes  for  Slide  2 


This  section  of  the  Code  of  Best  Practice  deals  with  issues  directly 
related  to  the  study  itself. 

Most  notably  the  Participants  within  the  project,  their  relationships 
and  the  dynamics  of  the  project. 

These  issues  can  be  as  interrelated  and  complex  as  the  subject 
under  study. 
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•  Roles  of  Participants 

•  Phases  of  a  Study 

•  Ethics  and  Conduct 


Topics 
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This  presentation  is  organised  into  three  parts: 

•  The  first  discusses  the  roles  played  by  the  significant  players 
associated  with  a  Command  and  Control  (C2)  assessment  and 
how  these  roles  affect  the  design  and  conduct  of  the 
assessment. 

•  The  second  part  identifies  the  major  phases  of  a  C2 
assessment  and  their  iterative  nature. 

•  The  concluding  section  addresses  the  subject  of  professional 
ethics  and  standards  of  conduct. 
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Roles  of  Participants 


•  Establish  who  is  Involved 

•  Establish  Roles 
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Like  their  subject,  the  organisation  of  C2  studies  involves  complex 
interrelationships. 

It  is  crucial  for  the  analytical  team  to  establish  which  individuals  and 
organisations  are  involved  at  an  early  stage  of  the  study. 

It  is  prudent  for  the  analytical  team  to  map  the  individuals  and 
organisations  involved  onto  the  roles  within  the  project  and  to 
understand  their  interrelationships. 
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The  identification  of  who  is  and  will  be  involved  in  a  project  is  a 
study  in  its  own  right. 

Due  to  the  dynamic  nature  of  such  projects,  those  involved  should 
not  be  surprised  if  the  nature  of  the  teams  involved  might  have  to 
expand  or  change  with  time. 

On  the  slide  is  an  example  of  the  organisations  involved  during  the 
recent  Immediate  Reaction  Task  Force  (Land)  (IRTF(L))  C2 
Concept  development  and  experimentation  project  that  completed 
at  the  end  of  2001 .  We  shall  use  this  to  illustrate  some  of  the  roles 
involved  in  a  complex  C2  Study. 
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The  assessment  team  is  working  for  the  sponsor  or  client. 

The  assessment  team  consists  of  a  senior  team  leader,  a  core  set  of  analysts,  subject 
matter  experts  including  military  officers,  and  supporting  staff  who  are  working  on  the 
study  on  a  day  to  day  basis.  The  team  provides  the  legitimacy  and  authority  for  the  study. 
In  the  case  of  the  IRTF(L)  study  the  assessment  team  was  led  by  NC3A  OR  Division,  with 
contracted  experts,  analytical  and  military  support  from  KS  Consultants  and  UK  DERA. 
Free  analytical  support  was  also  made  available  at  peak  periods  (such  as  during 
exercises)  from  US  JFCOM  and  SACLANT. 

The  sponsor  will  provide  the  terms  of  reference,  access  to  needed  information,  and 
identify  the  desired  products.  In  the  case  of  the  IRTF(L)  study  the  sponsor  was  SHAPE 
Policy  and  Requirements  Land  -  who  were  tasked  with  the  evaluation  of  the  C2  Concept. 

It  is  important  for  the  analytical  team  to  understand  exactly  why  the  sponsor  wants  the 
study  and  what  the  sponsor  wants  to  do  with  the  results.  In  the  case  of  the  IRTF(L),  this 
was  clear  for  most  of  the  project  -  a  straightforward  evaluation  of  the  military  utility  of  the 
C2  Concept.  However  right  at  the  end  of  the  study  the  results  were  combined  with  other 
issues  and  decisions  were  made  on  the  future  of  the  unit  under  study  -  something  that 
was  not  foreseen  at  the  start  of  the  project. 
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ORGANIZATION 

The  decision  makers  are  the  individuals  or  organisations  that  are  expected 
to  make  decisions  wholly  or  partially  based  on  the  output  or  findings  of  the 
study.  It  is  therefore  important  for  the  assessment  team  to  understand 
exactly  what  type  of  assessment  the  decision  makers  want  the  study  to 
support. 

The  decision  makers  may  or  may  not  be  in  command  of  or  part  of  the 
sponsor’s  organisation. 

In  the  case  of  the  IRTF(L)  study,  the  sponsor’s  task  was  to  provide  advice 
up  the  chain  of  command  to  SACEUR  and  ultimately  the  Military 
Committee  on  the  efficacy  of  the  C2  Concept.  The  ACE  Mobile  Force 
(Land)  was  the  subject  of  the  study  but  also  party  to  any  decision 
regarding  its  own  future  modernisation.  It  is  commanded  directly  by 
SACEUR  via  SHAPE.  SACLANT  CDE,  in  Norfolk  Virginia  however,  was 
not  in  the  command  chain.  It  was  only  interested  in  the  experience  of  the 
team  in  conducting  the  study  as  a  test  case  to  illustrate  the  value  of  NATO 
Centred  CDE  to  the  Alliance. 
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The  stakeholders  are  the  persons  or  organisations  that  are  directly  or  indirectly  affected  by  the  study  outcome. 

Stakeholders  may  also  play  other  roles.  The  assessment  team  must  be  aware  of  the  potential  for  conflict  when 
the  stakeholders  do  not  include  the  sponsor  or  decision  maker.  Complex  problems  may  arise  in  the  provision  of 
data  for  the  study,  as  it  is  the  stakeholders  who  will  often  have  to  provide  the  data,  set  the  security  or  releasibility 
of  that  data  (and  hence  the  study),  and/or  agree  that  the  data  are  representative.  For  these  reasons  it  is 
essential  that  the  analysts  establish  a  working  relationship  with  the  stakeholders  early  in  the  process. 

In  the  case  of  the  IRTF(L)  study  the  ACE  Mobile  Force  (Land)  was  the  main  stakeholder  as  it  was  the  subject  of 
the  study.  As  a  decision  maker,  data  and  assumption  provider  and  also  possible  member  of  a  future  study  team 
it  was  in  a  very  powerful  position,  and  was  approached  and  treated  with  much  respect  by  the  assessment  team. 
After  a  shaky  start  (where  neither  side  was  sure  of  the  other’s  intentions)  a  good  working  relationship  was 
established  over  the  period  of  the  project. 

The  Netherlands  MOD  -  in  the  form  of  the  Royal  Netherlands  Army  -  was  the  provider  of  the  Command 
Information  System  (ISIS)  used  as  the  digitisation  vehicle  for  the  evaluation  of  the  concept.  As  such  it  was 
directly  affected  by  the  exercise  program  used  for  the  experimentation  and  also  naturally  interested  in  the  future 
of  the  concept  and  AMF(L). 

The  Military  Committee  was  also  a  stakeholder,  representing  the  Nations  of  NATO  that  contribute  troops  and 
staff  to  the  AMF(L)  -These  nations  would  be  directly  affected  by  any  decision  on  the  concept. 

ACE  Resources  at  SHAPE  were  also  a  stakeholder  as  they  were  required  to  sanction  and  organise  any  manning 
changes  proposed  for  the  HQ  -  including  the  temporary  additional  manning  required  for  the  evaluation. 
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The  bill  payer  is  the  organisation  or  individual  official  paying  for  the 
study.  It  is  important  for  the  assessment  team  to  know  the  level  of 
resources  available.  Bill  payers  will  normally  have  a  direct  interest 
in  the  outcome  of  the  study  and  may  be  one  or  more  of  the  other 
players. 

In  the  case  of  the  IRTF(L)  study  monies  were  mostly  provided  from 
the  slice  of  the  NC3A  Scientific  Program  Of  Work  owned  by 
SHAPE  PRL.  In  the  initial  stages  of  the  project  some  additional 
monies  were  also  provided  by  SACLANT.  Also  in  order  to  attend 
the  exercises  some  monies  also  had  to  be  sought  from  the  SHAPE 
Exercise  budget  to  pay  for  movement  of  the  CDE  observers. 

Contractual  authorities  have  the  legal  authority  to  let  contracts  on 
behalf  of  the  bill  payer.  In  the  case  of  the  IRTF(L)  study  this  was 
NC3A  itself. 
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The  assessment  team  must  be  aware  of  and  sensitive  to  the 
existence  of  teams  in  other  related  study  areas. 

Should  such  teams  exist,  the  assessment  team  should  endeavour 
to  exploit  the  work  done  and  available  skills  and  techniques.  Such 
external  teams  may  also  be  appropriate  for  membership  in  peer 
reviews. 

In  the  case  of  IRTF(L)  an  extensive  literature  search  was 
conducted  -  with  the  majority  of  recent  references  within  the  UK 
and  US.  Exploratory  trips  to  US  Battle  labs  and  UK  facilities 
revealed  the  current  state  of  knowledge  with  respect  to  measuring 
C2  performance  in  exercises  and  evaluating  new  C2  concepts.  In 
response  to  this  our  data  collection  methodology  was  based  initially 
on  the  Fort  Leavenworth,  US  Army  Research  Institute  ACCES 
method. 
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ORGANIZATION 

The  assessment  team  must  be  aware  of  and  sensitive  to  the  needs  of 
future  analyses  and  assessments.  Data  collection,  method  documentation, 
and  the  archiving  of  data,  methods,  models  and  results  are  fundamental 
responsibilities  of  all  professional  analysts.  Method  and  data  should  be  (as 
far  as  is  practicable)  disseminated  and  published. 

In  the  case  of  the  IRTF(L)  Study,  it  was  identified  at  an  early  stage  that 
there  could  be  future  related  projects.  In  particular  those  relating  to 
expeditionary  and  initial  entry  forces.  The  probable  NATO  organizations 
that  could  be  involved  in  such  studies  were  us  (NC3A),  SHAPE  PRL, 
AMF(L),  Multinational  Division  (Central),  Strike  Force  South  and  the  ACE 
Rapid  Reaction  Corps.  Of  course  there  would  probably  be  future  study 
teams  within  the  nations  -  but  these  plans  are  not  visible  to  NATO. 
Consequently,  as  we  are  very  likely  to  be  involved  in  such  future  work;  all 
data  was  archived,  it  has  been  routinely  written  up  during  the  project  and 
the  final  report  and  technical  notes  are  in  the  process  of  being  published. 
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ORGANIZATION 

Outside  experts  brought  in  to  look  at  the  work  and  provide  constructive 
criticisms  are  called  peer  reviewers. 

Peer  review  teams  could  be  composed  of  specialists  and  other  study 
teams  in  related  subject  areas  and  should  include  representatives  from  all 
key  disciplines  in  the  assessment. 

In  the  case  of  the  IRTF(L)  project  we  could  not  arrange  a  formal  Peer 
Review  of  the  solution  strategy  adopted.  The  only  thing  that  we  could 
achieve  was  through  submitting  the  problem  to  the  SAS026  panel  as  an 
example  for  testing  the  coverage  of  the  COBP.  This  yielded  some  practical 
advice  and  helped  the  assessment  team  better  understand  the  dynamics 
of  the  project. 

This  novel  use  of  a  NATO  SAS  panel  was  obviously  not  a  permanent 
solution  to  our  problem  (which  will  occur  again  in  the  next  projects)  - 
however  since  we  were  able  to  present  our  problem  to  a  group  of  neutral 
experts  from  the  nations  this  might  be  a  future  area  where  RTO  could 
provide  a  service  to  the  analysts  of  the  nations  and  NC3A  ?? 
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Data  providers  are  the  individuals  and  organisations  that  possess 
data  and  information  useful  to  the  assessment  team. 

Many  of  these  will  be  stakeholders. 

The  motivation  to  provide  data  to  the  study  must  be  developed  by 
the  analytical  team  and  the  sponsor. 

In  the  case  of  IRTF(L)  most  of  the  data  was  derived  from  NATO 
training  exercises  run  by  or  for  AMF(L)  (a  Stakeholder),  MND(C), 
SFS  and  Joint  Sub  Regional  Commands  South  East  and  South 
Centre.  In  all  cases  relationships  had  to  be  curried  to  allow  access 
to  the  HQ  and  Exercise  Control  for  the  exercise  observers,  and  for 
background  materiel  such  as  Standard  Operating  Procedures 
(SOPs)  etc.  Historical  data  for  the  study  was  also  provided  from  the 
UK  Historical  Branch  (Army)  -  which  was  approached  via  the 
contracted  UK  members  of  the  assessment  team. 
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Assumption  providers  are  those  individuals  or  organisations  that 
can  provide  “givens”  such  as  future  doctrine,  performance  data, 
force  mixes,  organisational  structures,  and  scenarios.  Creation  of  a 
positive  relationship  with  these  organisations  is  important  to  the 
study 

In  the  IRTF(L)  Study,  we  were  in  the  fortunate  position  to  actually 
be  one  of  those  bodies  -  through  our  and  our  sponsor’s 
involvement  in  the  NATO  Defence  Requirements  Review.  The 
owner  of  the  C2  Concept  however  was  HQ  AMF(L)  itself,  and 
therefore  remained  the  authority  as  to  its  conceptual  and  physical 
implementation. 
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In  some  C2  analyses,  where  data  must  be  extracted  from  real 
world  experiences,  exercises,  experiments,  and  wargames,  teams 
of  data  collectors  and  subject  matter  experts  will  be  required.  The 
identification  of  people  with  the  appropriate  background  and 
training  as  data  collectors  are  important  elements  of  such  studies. 

In  the  case  of  the  IRTF(L)  study,  data  was  largely  extracted  through 
observation  of  HQ  activities  during  exercises  and  team-in-the-loop 
wargames.  In  all  of  these  exercises  Subject  Matter  Experts  were 
used  to  observe  functional  and  cross  functional  activities  in  the  HQ. 
Most  of  the  military  SMEs  were  provided  by  Regional  Command 
AF  NORTH  and  its  subordinate  commands  across  Allied  Command 
Europe  (ACE).  Additional  data  collectors  were  also  provided  by  the 
German  University  of  the  Federal  Armed  Forces  and  US  JFCOM. 
UK  DERA  provided  military  analysts  to  lead  some  of  the  activities 
involved  in  capturing  the  HQ  processes. 
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ORGANIZATION 

The  relationship  among  the  assessment  team,  the  key  sponsor,  and  the 
stakeholders  is  of  paramount  importance. 

This,  perhaps,  more  than  any  other  single  factor,  will  influence  the  course  and 
success  of  the  effort.  Accordingly,  adequate  attention  needs  to  be  paid  to 
understanding  the  situation  facing  the  key  sponsor  and  stakeholders  as  much 
as  the  subject  under  study. 

The  assessment  team  should  be  also  aware  that  the  different  participants 
will  have  divergent  perspectives  and  may  have  divergent  agendas. 

Therefore  the  assessment  team  should  make  sure  that  they  understand  the 
background  of  the  individuals  involved,  their  organisational  settings,  roles  and 
responsibilities,  their  history,  and  their  current  situation. 

Contact  with  analysts  who  have  worked  with  this  sponsor  and  review  of  prior 
analyses  for  this  sponsor  facilitate  this  objective.  OR  Organisations  must 
therefore  routinely  “market  ”  their  skills  within  the  organisations  with  which  they 
will  be  working. 
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It  is  important  that  a  dialogue  with  the  sponsor  and  stakeholders  is  maintained  by  the  assessment  team  throughout  the  study.  As  there 
is  no  single  “language”  that  will  describe  the  study  problem,  it  is  important  to  spend  time  at  the  beginning  of  a  study  to  establish  a 
common  “language”  that  both  the  assessment  team  and  the  sponsor  and  stakeholder  can  understand.  This  point  may  seem  obvious  in 
a  NATO  setting  in  which  the  participants  speak  many  different  natural  languages.  However,  it  is  equally  important  in  a  single  language 
setting  because  common  words  and  phrases  have  different  meanings  for  different  organisations,  services,  and  even  individuals  within  a 
single  organisation. 

Regular  meetings  and  contact  will  minimise  misunderstandings.  From  a  professional  point  of  view,  Operations  Research  (OR)  and 
Operations  Analysis  (OA)  analysts  will  always  wish  to  inform  the  sponsor  and  stakeholders  of  key  developments  and/or  challenges  as 
the  study  unfolds.  Regular  and  routine  interactions  need  to  be  built  into  the  project  plan.  If  there  are  multiple  sponsors  and  stakeholders 
and  other  key  actors,  the  assessment  team  should  try  to  meet  them  jointly,  particularly  when  decisions  need  to  be  made.  Separate 
meetings  will  often  lead  to  inconsistent  guidance  and  will  place  the  assessment  team  in  a  position  of  trying  to  accommodate  differing 
interests. 

Terms  of  Reference.  A  good  term  of  reference  covers  goals,  scope,  products,  schedule,  and  resources.  These  will  determine  the  focus 
of  the  assessment  and  establish  limits  or  freedoms  granted  to  the  assessment  team  within  the  sponsor’s  and  stakeholders’ 
organisations.  Letters  of  introduction  and  instructions  to  actors  within  the  sponsor  and  stakeholder  organisations  may  also  be  useful. 

It  is  important  to  know  at  an  early  stage  in  the  project  what  the  products  of  the  study  are  to  be  used  for  by  the  sponsor  and  stakeholder 
organisations.  The  expected  end  product  will  set  the  tone  and  relative  importance  of  the  project  in  the  eyes  of  the  sponsor,  stakeholder, 
and  other  actors.  The  assessment  team  needs  to  establish  and  understand  the  products  needed  or  desired  by  the  sponsor  and 
stakeholder.  For  example,  a  study  could  be  used  to;  effect  a  significant  impact  on  the  stakeholder’s  domain,  gain  a  greater 
understanding  of  the  issues,  produce  an  improved  capability  to  perform  future  work,  and/or  make  contributions  to  the  body  of 
knowledge. 

The  sponsor  will  have  limited  resources  and  a  study  budget  in  mind.  When  the  sponsor’s  resources  are  limited  to  a  level  below  what  is 
required  to  support  a  quality  study,  the  assessment  team  will  need  to  suggest  strategies  to  address  the  shortfall.  Alternative  approaches 
include  decomposing  the  problem  and  only  undertaking  the  core  part  of  the  study  that  is  affordable,  linking  the  sponsor  to  other  actors 
that  have  an  interest  in  the  same  or  a  similar  problem  and  who  could  contribute  resources,  and/or  stretching  the  project  over  a  longer 
time  so  that  resources  from  future  budget  cycles  become  available.  In  developing  strategies  that  involve  doing  only  a  part  of  the  study  to 
satisfy  budget  constraints,  care  must  be  taken  to  ensure  that  the  product  that  will  be  produced  provides  a  meaningful  answer  or 
contribution  and  does  not  depend  upon  a  follow-on  effort  that  may  or  may  not  be  funded. 
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ORGANIZATION 

Moving  now  to  the  second  part  of  this  briefing.  We  will  now  look  at 
the  phases  of  a  study. 

It  is  important  to  realise  that  all  of  the  elements  of  the  C2 
assessment  are  interrelated.  Hence  [point ]  Problem  Formulation, 
Solution  Strategy,  Measures  of  Merit,  Scenarios,  Human/ 
Organisational  Factors,  Models/Tools/Data,  and  products  are  all 
interdependent.  This  diagram  illustrates  the  major  phases  and 
iterative  nature  required  for  C2  assessments.  This  diagram  was  the 
most  difficult  thing  for  the  SAS026  team  to  agree  on  and  produce. 
In  essence  it  is  at  the  heart  of  he  COBP. 

We  shall  now  discusses  the  key  points  in  this  diagram. 
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The  output  of  Problem  Formulation  specifies  the  “what”  of  the  assessment. 

C2  studies  tend  to  be  complex  and  feature  multiple  attributes,  some  of  which 
may  not  be  apparent  at  the  start  of  the  study.  Neither  the  assessment  team  nor 
the  sponsor  should  be  surprised  if  the  issues  initially  presented  for  study  are 
replaced  by  other  issues  that  are  closer  to  the  underlying  causes  of  the  initial 
problem  or,  in  some  cases,  symptoms  presented.  Problem  formulation  should 
therefore  be  re-performed  routinely  during  a  study  -  especially  when  new 
attributes  start  to  appear.  As  a  minimum  this  should  be  immediately  following 
the  establishment  of  the  solution  strategy  and  any  assessment  of  study  risk. 

A  consequence  of  the  dynamic  nature  of  problem  formulation  is  that  the 
solution  strategy  and  any  of  the  other  elements  of  the  assessment  may  also 
need  to  change  as  the  study  progresses. 

Additionally,  the  sponsor  should  be  aware  that  the  nature  of  the  assessment 
team,  sponsor,  or  assumption  provider  teams  might  also  have  to  expand  or 
change  with  time.  This  has  implications  for  planning,  budgeting,  and  tasking. 
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•  Core  Team 


Building  an  Assessment  Team 


-  Project  management; 

-  OR/OA  skills 

-  Cross  military  experience  *  Consultants  and  Part-time  Team 

-  Organisational  theory  -  Military  training  and  exercise  planning 


-  Data  collection. 


Expertise 

Requirements 


Communications  and  information 
systems 

Human  interfaces 

Operations  Other  Than  War  (OOTW) 
Social  science 
Military  history 
Command 

Deployment  analysis 
Intelligence/threat  assessment 
Legal/contracts/administration. 
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Following  initial  problem  formulation,  the  precise  skills  and  experience  required  by  the 
assessment  team  will  need  to  be  established  -  this  is  in  response  to  the  results  of  the 
problem  formulation. 

Typically,  the  assessment  team  will  need  to  be  interdisciplinary.  The  wide  range  of  skills 
and  experience  needed  can  be  allocated  between  a  full  time  core  team  and  a  collection  of 
consultants  or  part-time  team  members.  As  an  example,  this  slide  shows  an  ideal 
breakdown  of  the  skills  available  to  the  assessment  team  involved  in  the  evaluation  of  the 
IRTF(L)  C2  concept  study. 

Following  the  identification  of  the  skills  or  expertise  required  of  the  team,  those  analysts 
made  available  for  the  core  team  should  ensure  that  they  have  a  basic  understanding  of 
the  military  fields  under  consideration.  Gaps  in  experience  should  be  filled. 

Building  a  C2  assessment  team  with  this  full  breadth  of  knowledge,  capabilities,  and  skill 
requires  a  long-term  commitment  to  preparing  a  corpus  of  potential  team  members 
through  recruitment,  education,  training,  and  provision  of  opportunities  for  appropriate 
field  experience. 


Now.  It  is  one  thing  to  assemble  a  group  of  people,  quite  another  to  forge  them  into 
a  coherent  and  effective  team.  Time  should  be  allowed  for  the  group  to  form  into  a 
team. 
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The  next  step  is  to  develop  a  Solution  Strategy  that  specifies  the  “how”  of  the 
assessment. 

Once  the  Solution  Strategy  has  been  agreed  and  adopted  by  the  sponsor  a  set  of  terms 
of  reference  (e.g.  Statements  of  Work  [SOW]  for  contracts)  should  be  produced  that  will 
determine  what  work  is  to  be  conducted,  the  contractual  obligations,  deadlines,  and 
resources. 

Although  these  must  be  established  as  a  project  plan  before  work  on  the  project  begins  in 
earnest,  flexibility  must  be  built-in  due  to  the  iterative  nature  of  C2  assessment. 

The  assessment  team  must  be  aware  of  any  preconceived  “solutions”  that  have  been 
proposed  by  the  sponsor,  stakeholders,  and/or  decisionmakers  and  explicitly  deal  with 
these  as  appropriate,  avoiding  another  pressure  to  be  steered  in  a  particular  direction. 

The  assessment  team  also  must  note  if  its  results  are  being  steered  in  a  particular 
direction  and  follow  ethical  behaviour  in  performing  the  study. 

In  many  cases  a  risk-based  approach  to  C2  assessments  can  usefully  complement  the 
more  traditional  cost-effectiveness  approach.  In  particular,  this  will  help  decisionmakers  to 
deal  with  the  uncertainties  of  the  real  problem. 
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Once  there  has  been  a  preliminary  formulation  of  the  problem  and 
development  of  a  solution  strategy,  it  is  imperative  that  an  initial 
review  be  conducted. 

This  review  should  be  conducted  from  multiple  perspectives 
(e.g.  with  respect  to  the  sponsor’s  initial  problem,  the  feasibility  with 
respect  to  resources  including  team  skills  and  schedule,  soundness 
of  the  proposed  analytic  approach). 

As  a  result  of  this  review,  changes  will  usually  be  made  in  both  the 
problem  formulation  and  the  solution  strategy. 
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We  now  enter  a  period  in  a  C2  study  where  the  tasks  are 
inextricably  interlinked. 

At  this  stage  the  assessment  team  must  specify  the  hierarchy  of 
Measures  of  Merit  (MoM),  incorporate  and  identify  relevant  human 
and  organisational  factors,  and  specify  the  appropriate  scenarios. 

As  suggested  by  the  diagram,  there  is  no  unique  sequence  for 
doing  these  tasks. 

Iteration  is  required  to  ensure  that  these  tasks  are  done  in  a 
coherent,  consistent  fashion. 

When  all  of  these  tasks  have  been  completed,  the  team  will  have 
specified  the  key  variables  to  the  necessary  level  of  detail  with 
adequate  considerations  for  assessment  validity  and  reliability. 
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We  now  enter  another  period  in  a  C2  study  where  the  tasks  are  inextricably  interlinked. 

The  next  step  is  to  iteratively  identify  the  methods  and  tools  and  data  required  to  perform  the 
assessment.  One  of  the  major  challenges  of  the  assessment  is  to  identify  and  gain  access  to 
models,  tools,  and  data  that  are  appropriate  for  exploring  the  issues  of  interest. 

The  challenges  come  in  several  dimensions: 

-  First,  there  is  a  limited  set  of  tools  that  deal  effectively  with  the  C2  dimension  of  the 
problem. 

-  Second,  for  even  this  limited  set,  it  is  often  difficult  to  access  and  modify  the  tools  to  reflect 
the  variables  of  interest. 

-  Third,  there  is  often  a  paucity  of  useful  data  and  previously  validated  parameters. 

As  a  result  of  the  establishment  of  the  MoM  for  the  study  and  the  data  that  underpins  those  MoM 
and  models,  a  data  collection  and  analysis  plan  should  be  formulated.  The  sponsor  should  also 
be  made  aware  of  the  difficulties  associated  with  getting  appropriate  data,  cost  of  the  data 
collection  and  analysis  plan,  and  the  implications  to  the  study  if  the  required  resources  are  not 
set  aside  and  budgeted  to  collect,  collate,  process,  and  analyse  the  data 
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At  this  point  in  the  process  the  assessment  team  should  take  a  look 
at  the  risks  and  uncertainties  associated  with  the  decisions  made 
with  respect  to  all  of  the  tasks  performed  to  date  (e.g.  consistency 
between  the  scenarios  and  the  data,  models  and  availability  of 
data,  tools  and  analysis). 

The  sponsor  must  be  made  aware  of  these  risks  and  uncertainties 
and  the  strategies  developed  by  the  team  to  mitigate  them. 

If  the  risks  associated  with  the  successful  completion  of  the  study 
are  perceived  as  being  too  high,  the  solution  strategies  should  be 
revisited  and  adjusted  accordingly. 
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When  the  risk  and  uncertainties  are  perceived  as  manageable,  a  peer 
review  should  be  conducted. 

Peer  reviews  are  not  used  enough  because  they  tend  to  be  time- 
consuming,  seen  as  raising  costs,  or  perceived  as  threatening.  In  addition, 
research  teams  often  want  to  perfect  their  results  and  methods  before 
revealing  them. 

The  key  is  to  build  a  peer  review  into  the  study  from  the  outset.  The 
sponsor  should  be  informed  as  to  the  importance  of  the  peer  review.  Peer 
reviews  should  be  built  into  the  budget  and  reviewers  invited  to  look  at  the 
terms  of  reference,  interim  products,  and  draft  reports  so  that  they  are 
knowledgeable  about  the  effort  and  motivated  to  support  the  project.  In 
later  stages  of  the  study,  the  peer  review  can  improve  presentation  and 
also  act  as  a  mechanism  to  make  the  results  known  to  the  professional 
communities. 

Over  time  the  assessment  team  should  develop  a  relationship  with  high 
quality  peers  and  use  them  as  a  pool  of  reviewers. 
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At  this  point  we  are  in  a  position  to  execute  the  assessment. 

The  assessment  team  leader  should  keep  a  study  notebook  or  journal  in  which  all 
assumptions  and  decisions  are  documented  so  that  they  are  available  for  detailed 
discussion. 

Detailed  administrative  records  need  to  be  kept  regarding  the  data,  metadata, 
models,  and  analytical  and  documentation  tools.  This  will  enable  replication  of  parts 
of  the  C2  analysis  should  the  need  arise. 

An  effort  should  be  made  to  create  data  sets  (not  just  the  project  results)  that  will  be 
available  to  other  researchers.  The  resources  required  to  make  such  data  available 
to  external  bodies  needs  to  be  made  clear  to  the  sponsor. 

The  conduct  of  the  study  will  not  usually  be  linear.  It  should  be  anticipated  that 
multiple  iterations  will  be  conducted  and  that  lessons  learned  from  initial  data 
collection  and  analysis  efforts  will  inform  subsequent  activities. 

The  team  must  recognise  the  importance  of  presenting  the  results  of  the 
assessment  in  a  clear  and  comprehensive  manner,  taking  into  consideration  the 
style  of  the  decisionmaker.  It  is  particularly  important  that  these  results  illuminate 
rather  than  obscure  the  uncertainties  associated  with  the  assessment. 
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•  Open  and  honest  dialogue 

•  Conduct  balanced  C2  assessments 

•  Notify  Sponsor  and  key  players 

-  Threats  to  balanced  assessment 

-  Risks  and  uncertainties  of  data  and  methods 

-  Strategies  to  minimise  risks 
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In  the  final  and  third  part  of  the  presentation  I  shall  briefly  touch  on  ETHICs. 

Professional  operations  research  organisations,  such  as  the  Military  Operations  Research 
Society  (MORS)  and  the  European  Operations  Research  Society,  have  developed 
professional  codes  of  ethics. 

The  assessment  team  should  also  be  guided  by  a  set  of  professional  ethics  and 
standards  of  conduct  to  ensure  the  integrity  and  quality  of  the  analysis.  This  means  that 
the  assessment  team  should  -  if  we  take  the  MORs  Ethics: 

•  Maintain  an  open  and  honest  dialogue  with  the  sponsor  and  other  key  players  within 
the  project  in  order  to  minimise  misunderstandings; 

•  Ensure  that  C2  assessments  are  organised  and  conducted  in  a  balanced  fashion 
that  adequately  identifies  and  represents  all  perspectives,  options,  and  relevant 
evidence; 

•  Inform  the  sponsor  and  other  key  players  of: 

•  any  constraints,  assumptions,  or  circumstances  that  threaten  a  balanced 
assessment; 

•  the  risks  and  uncertainties  associated  with  the  methods  and  data  used  in  the 
project, 

•  strategies  to  minimise  these  risks. 
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